Towards Comprehensive Web Search

Towards Comprehensive Web Search by Erik Warren Selberg Chair of Supervisory Committee Associate Professor Oren Etzioni Department of Computer Science & Engineering The World Wide Web has rapidly become a key medium for information dissemination to all members of society. However, its disorganized nature and sheer size can make it difficult for people to find information. Web search services have made a significant contribution towards enabling people to quickly find information on the Web. Unfortunately, as of this writing, no Web search service can conduct a comprehensive search of the Web for any topic. In addition, many major Web search services are unable to return a stable set of results. An intuitive assumption about the behavior of any search service is that the results of a given query will be unchanged unless either the documents referred to in the results change and become irrelevant or better documents become available. Unfortunately, due to a variety of real-world constraints and design choices, many search services are unstable, intermittently omitting relevant documents from search results even though the documents are contained within their indices. One technique that could enable both a more comprehensive search of the Web as well as a more stable search is meta-search. Meta-search is conducting a single search using multiple search resources. This thesis examines the application of meta-search to the World Wide Web. It answers the following questions: can meta-search provide a more comprehensive search than traditional Web searching? Is meta-searching necessary now, and will it be necessary in the future? Can meta-searching be implemented in a practical manner? And can meta-search enable a more stable search? We present MetaCrawler, a meta-search service, as a means of obtaining a more comprehensive search than existing Web search services. MetaCrawler addresses some of the issues with Web search service through forwarding a user query and combining the results from multiple search services into a single list. To summarize the results of this thesis, we conclude that MetaCrawler demonstrates that meta-search can be implemented in a manner such that average Web users will take advantage of the benefits of meta-search. We also conclude that MetaCrawler demonstrates that meta-search can be provided and maintained with limited resources. Through our experiments using Inference of User Value through Real-world data, a new methodology to evaluate search services, we conclude that MetaCrawler provides a significantly more comprehensive search than any single search service. Furthermore, the growth trends of the Web and search service indices lead us to conclude that meta-search will be necessary to provide a comprehensive search in the future. Our experiments also demonstrate that most major search services are unstable, and may omit relevant documents even if those documents are present in their indices. Finally, we conclude that search results can not only be made stable, but can be improved through Collaborative Index Enhancement, a novel model for enhancing a searchable index based on the experience of previous users.